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Methodology 



The author was asked to review the following six documents and to summarize any explicit 
or implicit recommendations in them which were relevant to the charge of the Advisory Committee 
on Testing in Chapter 1: 

Council of Chief State School OfBcers, Hawkins-Stafford Reauthorization Task Force, 
Working Paper Reauthorization of the HawtdnsStafford Amendments of 1988, 
September 1, 1992. [CCSSO] 

Deich, S.G., Sherman, J.D., Amstutz, W. and Shifflnan, J., Summary of Public 
Comments Regarding the Reauthorization of Elementary and Secondary Education 
Programs, Pelavin Associates, Inc., April 30, 1992. [SPC] 

Kean, M.H., ESEA Chapter 1 Reauthorization: Testimony Before the Advisory 
Committee on Testing in Chapter 1, Association of American Publishers, August 12, 
1992. [Kean] 

Padia, W.L., Chapter 1 Assessment Issues, California Department of Education, 
August 12, 1992, [Padia] 

U S Congress, Office of Technology Assessment, Testing in American Schools: Asking 
the Right Questions, OTA-SET-519 (Washington, DC: U.S. Government Prmtmg 
Office, February 1992). [OTA] 

U.S. Department of Education, Office of Management and Budget, Regional Hearing 
Summaries, April 27, 1992. [RHS:City] 

The results of this review are presented in three parts below. First there is a brief description 
of a framework for discussing different uses of assessment Second, specific recommendations from 
each of the above six sources are presented. Each recommendation is linked to its source by a code 
enclosed in square brackets, e.g., [OTA-89]. The code for each source can be found at the end of 
the citations listed above. The number attached to each code represents the page of the do^ment 
from which the recommendation was taken, e.g., [OTA-89] indicates page 89 from the Office of 
Technology report. 

The selection of recommendations was based on the author's judgement These 
recommendations should not be taken as direct quotes, although most are very close to the ongmal 
language. The authors of the original documents might not agree with the selection or wordmg ot 
what is presented here. An attempt was made to organize the recommendations into a useful set of 
categories. Placement in some cases was arbitrary. Occasionally, a single recommendation may 
appear in more than one category. 

FoUowing the specific recommendations is a list of summary recommendations created by the 
author based on what the specific recommendations seem to be saying. Others might not interpret 
the specific recommendations in the same way. These summary recommendations are numbered only 
to facilitate discussion. 



Framework for Discussing Uses of assessment 



Gri^r.\l U -^^" "f Assessment 

The uses of educational tests can be grouped into three areas: [OTA-10] 

. to aid teachers and students in the conduct of classroom learning 
• to monitor systemwide educational outcomes 

. to inform decisions about student selection, placement and credentialmg 
See Table 6-3. [OTA-195] 
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Chapter 1 U^sFj; of Assessment 



Author's Alignment of Chapter 1 Uses with OTA's Framework 
S y;tem Monitoring 

• national accountability (TIERS) 

• state evaluation (biennial report) 

• local program evaluation 

• progress in the regular program 

• schoohvide project accountability 

• determine need for school program improvement plan 

• determine levels of need for Chapter 1 services in schools, grades 
and subjects 

Selection and Placement 

• identify and select eligible students 

• identify students not progressing adequately 

Instructional Guidance 



• diagnosis of individual student needs 
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Specific Recommendations 
Regarding Uses of Assessment 



GEffERAL Chapter 1 Use 

Allow SEAs and LEAs to use a variety of assessment methods that reflect their own goals for 
Chapter 1 in order to modify programs, place students, target schools for program 
improvement, decide on continuation of schoohvide projects, etc [OTA-34] 

Reduce the reliance of Chapter 1 assessment and evaluation on NRTs, [OTA-90] 

Reduce the amount of testing required. [OTA-90] 

Decrease emphasis on NRTs by scaling down the roles testing plays in Chapter 1 programs, 
reducing the number of areas and students tested, and reducing the frequency of testmg. 
[RHS:LA-2] 

Give states the flexibility to taQor their assessment strategy to the requirements for selection, 
needs assessment, identification of students, and evaluation- Each state's assessment plan 
could be negotiated with USDE. [Padia-5] 

Chapter 1 AccouNTAarLrrY/EvALUATioN 
General 

National and state/local Chapter 1 program administrators' data needs are different and not 
necessarily well met by NRTs. [OTA-89] 

For accountabiUty, multiple assessments should be used to determine student achievement and 
program effectiveness. [CCSSO-18] 

There are existing state assessment systems that should be taken advantage of, and state 
should be charged to develop better systems for accountabihty. SEAs should be required to 
have an assessment system which includes a mix of assessments from which to make decisions 
about students and programs. [CCSSO-19] 

NRTs should not be used for accountability or selection. [RHS:DC-3] 

Should assessment of Chapter 1 students be requked annuaUy and in each grade? (7 yes - 

4 no) lSPC-50] 

• Yes - for determining special needs. 

• Yes - give quarterly. 

• Fall to spring is better. 

• Yes - if have to have gain scores. 
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Should student assessment criteria other than standardized tests be used? (10 yes - 0 no) 
(SPC-51] 

• Would like to see improvement in the classroom as the mam measure of success. 

Demonstrate student performance by supplementing NRTs with alternatives like writing 
assessment [Kean-16] 

Consider technical as well as policy implications of comparing or equating different assessment 
instruments even if all the instnmients are aligned to the same content standard. [Kean-18] 

Do not require matched pretest and posttest scores. High mobility creates potential for 
unrepresentative results. [Padia-6] 

National AccouNXABnjTV 

Conduct separate national assessment of Chapter 1 which would allow sampling, less frequent 
testing for students, less time spent testing by teachers and other school personnel, and more 
control over data quality. [OTA-35] 

The national component of a state's assessment system would include standardized criteria 
or NRTs using a statewide sample (or as part of a national sample) for program 
accountability. [CCSSO-20] 

Test a sample of Chapter 1 students. [RHS:LA-2] 

Administer tests less frequently at specific grades with measures of performana?. 
[RHS:Seattle-3] 

Should testing be aligned with national assessments measuring performance less frequently 
and at specific grade levels? (4 yes - 2 no) [SPC-51] 

. Yes - use NRT data at grades 4, 8, and 12. 

• No - national assessments are not uniform, they are voluntary'. 

Allow statewide sampling of Chapter 1 programs and student outcomes in order to provide 
national comparisons. [SPC-52] 

Obtain data for a national evaluation through a national sampling at selected grade levels, 
perhaps through NAEP or another national testing system. [Padia-7] 

Tests used in a national evaluation should incorporate authentic, performance-based measures 
and should yield the percentage of students achieving specified performance standards. 
[Padia-7] 



5 

ERIC 



Evaluating State/Local Program Effectiveness 



Consider USDE funding of research and development on better assessment practices at the 
state and local level. [OTA-34] 

The identification of schools for program improvement and the evaluation of schooiwide 
projects has required more of NRTs than they can, perhaps, provide while also attaching 
greater consequences to their results. [OTA-89] 

The state component of a state's assessment system would include district or state developed 
CRTs (including performance-based assessments) based on state outcomes and standards of 
performance. [CCSSO-20] 

The local component of a state's assessment system would include other indicators of 
performance which LEAs and schools could use to measure individual student achievement 
such as portfolios, observed performance and participation rates. [CCSSO-20] 

Require states to develop their own assessment techniques with stringent outcome measures 
and performance standards that could be aligned with state and local assessment practices. 
[RHS:DC-3] 

Work with SEAs and LEAs to develop viable alternatives to NRTs to determine outcomes 
of Chapter 1 programs. [SPC-52] 

Use multiple measures which reflect the breadth and depth of desired outcomes. [Kean-15] 

SEAs and Committees of Practitioners should be responsible for setting state standards of 
performance for Chapter 1 programs at selected grade levels, with attainment measured by 
state or nationally developed tests. Permit and encourage tests with strong performance 
assessment components. Encourage LEAs to set standards for other grades. Allow NRTs 
until high-quality performance measures are available. Encourage the use of multiple 
indicators. [Padia-9] 



DE^ERMI^aNG Need for School Program Improvembxt Pian 

The identification of schools for program improvement and the evaluation of schoohvide 
projects has required more of NRTs than they can, perhaps, provide while also attaching 
greater consequences to their results. [OTA-89] 

A better match is needed between the goal of improving the quality of local Chapter 1 
programs and the tools used to measure progress toward that goal [OTA-89] 

Modify Chapter 1 program improvement requirements to allow other desired outcomes to 
determine buildings required to implement program improvement plans. [SPC-52] 
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Should the use of a particular measure of program eEfectiveneas be required for identifying 
schools for program improvement? (4 yes - 6 no) [SPC-59] 

• Writing samples, portfolios, basal unit tests and classroom teacher surveys should 
be allowed for measuring substantial progress toward desired outcomes. 

• Multiple indicators (NRTs and desired outcomes) should be used for targeting 
program improvement sites, using the preponderance of evidence approach. 

. Allow use of other desired outcome in conjunction with aggregate performance. 
. LEAs should have the flexibility to choose between standardized testing and 
desired outcomes. 

Should student assessment criteria, other than stand&fdized tests, be used for program 
improvement? (9 yes - 0 no) [SPC-64] 

• Teacher evaluation should be used 

. School districts should design their own assessment plans. 

• There should be a grea ter emphasis on desired outcome measures. 

• Performance assessmeni should be used, 

A local committee should set goals, standards, and measures for determining the success of 
the Chapter 1 program. Results should be reported to the LEA and SEA. SEA should 
provide technical assistance in skills of settmg high and reasonable goals and standards and 
selecting appropriate measures and instructional practices. SEA should provide mcentives for 
LEAs taking on high goals and innovative practices. [SPC-65] 

A broad view of assessment is needed for gauging improvement Training and monitoring arc 
important in setting performance standards. [SPC-66] 

Consider most appropriate ways to measure progress and determine the need for program 
improvement, such as supplementing NCE gains with other measures and judging Chapter 1 
students relative to other Chapter 1 students instead of aU students. [Kean-18] 

Minimize disincentives for setting high standards and teaching higher order skills. [Kean-18] 



DETEJUvaNiNG Levels of Need for Services 

Funding should be stable for Gve years instead of a function of test scores. [RHS:Seattle-3] 



Other Chapter 1 Uses of Assessment 

iDENTIFICATION/SELECnON 

Tests used for selection or placement should be designed and vaUdated for that purpose. 
[OTA-185j 

Tests designed to be used as feedback mechanisms to inform the learning process [referring 
to NUTs] should not be used to make significant decisions about an individual's educational 
career unless additional evidence c?n be provided substantiating this use. [OTA-185] 



Let the teachers decide who is in the Chapter 1 program. Use portfolios to determine 
students' abilities. [RHS:DC-3] 

Methods of identifying Chapter 1 students should be modified - alternatives to NRTs should 
be developed and utilized. [SPC-14] 

De-emphasize NRTs for pre-Kindergarten through 3rd grade as a means for selecting 
students for Chapter 1. Emphasize the use of developmental assessments, readiness tests, 
portfolios, checklists, and docuncntcd teacher observations. [Padia-S] 



Instructional Monitorinc 

One of the dangers in relaxing technical standards for classroom tests is that the use of the 
scores cannot be restricted or monitored appropriately once they are obtained. [OTA-196] 

Provide for periodic monitoring of student progress. [Kean-15] 



GENEt^AL ASSESSKfENT ISSUES 

Relationship of Assessment to Cuwuculum 

Stimulate the development of assessment methods more suitable to the goals of Chapter 1. 
[OTA-90] 

Avoid requirement to measure achievement with NRTs which result in schoo'/s deUvering a 
low-level skills-based curriculum to disadvantaged students. [RHS:LA-2] 

NRTs measure isolated versus complex skills. [RHS:Seattle-3] 

Should Chapter 1 assessment practices be aUgned with current movement to develop national 
standards for all children? (yes/no count missing) [SPC-48] 

• Should focus on advanced skills and complex tasks. 

• Develop guidelines for portfolio assessment to supplement NRTs. 

• Should be aligned with local assessment 

• Allow for change as state and national standards are pu in place. 
. Identify Chapter 1 students in NAEP and measure gap. 

• Emphasize other desired outcomes as much or more than NRTs. 

• Allow states to set standards for other desired outcomes. 

Develop and expand training and technical assistance to LEA teachers and administrators to 
improve the link between instruction and assessment [SPC-521 

Articulate assessment information to national education goals, national curriculum standards, 
and to state and local assessment approaches. [Kean-15] 

Continue to assess both basic and higher order skills. [Kean-16] 
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Good Testing Practices 

Emphasize adherence to good testing practices, including vaUdity, reliabiUty, fairacss, security, 
especially for uses outside the classroom. [Kean-15] 

Use different tests for different purposes. [Kean-15] 

Present assessment results in a readily understandable format [Kean-15] 

Consider consequences of high stakes testing independently of test format [Kean-181 

Monitor the use of tests for appropriateness. [Kean-18] 

Take care that the use of alternative assessmenU do not create test bias problems. [Kean-lS] 
Special Populations 

Special attention must be given to assessment results for homeless, migrant and N or D 
children. [CCSSO-18] 

Use of avaUable date from tests in chUd's nati%'e language is necessary in some circumstances. 
[CCSSO-18] 

Wait until the end of 2nd grade before administering tests. [RHS:LA-2] 
Omit 2nd and 3rd grades from standardized testing. [SPC-52] 

Performance Measures 

Careful development of scoring criteria and intensive training of judges are the key to 
establishing consistency of judgment [OTA-242] 

Wlien only a few tasks are used there is a much higher risk that a chUd's score will be 
associated with the particular tasks and not generalize to the whole subject area that the tes 
is meant to cover. This can be mitigated by sampling students and tasks if scores arc not 
required for every student on every task. [OTA-242,243] 

Benefits in curriculum and staff development may ofi&et the higher costs associated with 
performance assessment [OTA-245] 

Data on the impacts of performance assessment on various subgroups is needed in conwdering 
whether to employ such measures in high stakes situations such as school accountabdity or 
student selection. [OTA-247] 
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If performance assessment is given a larger role in testing programs, teachers will need to be 
involved in designing tasks, administering and scoring tests, and placing test results into 
context [OTA-248] 

Writing assessment is now workable for all three major testing functions. Other methods of 
performance assessment (e.g., portfolios, exhibitions, experiments, and oral interviews) still 
represent relatively uncharted areas. They have potential for classroom instructional guidance 
and system monitoring through sampling. But much research is necxled before they can be 
used for high-stakes applications in students selection and placement [OTA-249]. 

Because performance assessment is at a developmental stage, encouraging states and districts 
to pool experience and resources is an appropriate policy goal [OTA-249] 

Augment NRTs with performance assessment, doing so gradually and give teachers a chance 
to buy into it [RHS:LA-2] 

New TECHNOLocrES Foa Assessment 

The Federal Government could continue to support research and development of a wide 
range of new models for testing in various ways, including earmarking resources in programs 
like Chapter 1 for research into how advanced technologies can improve testing. [OTA-279] 

Use emerging technology to support periodic monitoring of student progress and use this 
information to improve instruction. [Kean-15] 
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Summary Recommendations for Chapter 1 Assessment 



1 DctennLie what Congress, national, state and district Chapter 1 administrator, teachers, 
parents, and communities need to know about student performance and knowledge. 

2. Identify the appropriate types cf assessment (e.g., tests) that are designed to provide the 
different types of information these groups need. 

3 Administer these appropriate assessments to the minimum number of students classrooms, 
^aSr^^ooU. or^Ltricts that arc needed to provide leliable informaUoo. For o^pk^ 
USDE may not need information about students at every grade level, but distncts might need 
informaUon for every grade in which Chapter 1 services are provided. 

4. Sampling should be used wherever appropriate and technically feasible, 

5. Reduce the reliance of Chapter 1 assessment on current norm-referenced, standardized, 
multiple choice, achievement tests. 

6 Performance-based measures should be used where feasible and appropriate since Jhey 
provide a more vaUd assessment of student outcomes associated with curricula based on 
recent research and theoretical developments in cognitive psychology. 

7. Assessment to meet accountability needs should be conducted at a national level by the 
Federal Government. 

8. Assessment to determine where and how to improve Chapter 1 services should be conducted 
at the state and local level by SEAs and LEAs. 

9. National accountability assessment should be linked to the national goals and curriculum 
Standards. 

10 State and local assessments the effectiveness of Chapter 1 services for the purpose of 
improrg these services should be based on state and local desired outcomes and cumculum 
Standards. 

Some states and school districts will need assistance in setting reasonably high goals or 
o^^m« constructing appropriate standards, and selecting or developmg appropriate 
measures. 

12 Assessment used to identify and select students for Chapter 1 services should be based on 
* multiole measures and linked to the state and local desired outcomes and standards. 



11. 



13. 



Assessment to diagnose individual student needs and monitor student progress in the 
classroorn should be closely linked to the instruction provided by the teacher. 
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14. Use different types of assessment only for the purposes for which they have been shown to 
be reUable (consistent), valid (accurate), and unbiased (fair). 

15. Present assessment results in a format that is easily understood by the audience for which they 
are intended. 

16. Assessment results used to make high stakes decisions should be based on multiple measures. 

17 Assessment methods should be designed to accommodate, rather than exclude, special 
populations, e.g., limited English proficient, homeless, early grade levels, mi^t, and 
neglected or delinquent 

18 Developers of performance-based assessments should involve teachers in designing tasks, 
administering and scoring performance measures, and interpreting results for intended 
audiences. 

19. Provide support for the examination and development of new technologies for assessing 
student performance and knowledge. 
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